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From  the  Director 

As  I  write  this  article,  two  things  come  to  mind  that  1  want  to  communicate  to 
our  newsletter  recipients.  The  first  is  the  goal  of  the  U.S.  Army  Engineer  and 
Research  and  Development  Center  (ERDC)  Major  Shared  Resource  Center 
(MSRC)  (through  the  Department  of  Defen.se  (DoD)  High  Performance 
Computing  Modernization  Program  (HPCMP)),  which  is  to  deliver  high 
performance  computing  (HPC)  leadership,  service,  education,  and  technical 
expertise  to  achieve  research  and  engineering  objectives  vital  to  the  Nation. 
Helping  to  provide  the  U.S.  warfighter  the  advantage  on  the  battlefield  by  means  of  HPC  is  of  utmost  importance 
to  this  Center.  The  largest  impact  that  our  Center  can  have  to  accomplish  this  is  by  providing  our  u.sers  the  latest  in 
HPC  and  by  striving  to  make  the  effective  use  of  these  resources  easier. 

The  second  thing  is  recognizing  the  importance  of  helping  to  shape  the  future  with  outreach  efforts  in  educational 
institutions  to  boost  science  and  engineering,  thereby  helping  to  enable  the  United  States  to  regain  leadership  in 
technology.  According  to  Warren  Washington.  Chairman  of  the  National  Science  Board,  which  oversees  the 
Federal  National  Science  Foundation,  our  country  is  in  a  race  to  retain  its  global  advantage  in  .science  imd  engi¬ 
neering. 

Much  of  the  information  that  we  have  chosen  to  publish  in  this  edition  of  our  newsletter  supports  my  thoughts 
above.  Day-to-day  challenges  that  our  Program  faces  must  not  overshadow  our  purpose  here  at  the  ERDC  MSRC. 
With  our  dedicated  staff,  we  will  continue  striving  to  provide  our  users  with  the  latest  technology  and  the  help 
through  training  and  innovative  solutions  by  our  staff  to  use  it.  We  will  continue  with  our  outreach  effort  to  attract 
the  best  and  brighte.st  students  to  science  and  engineering  and  retain  them  to  ensure  technological  leadership  for 
the  next  generation. 


John  West 

Director,  ERDC  MSRC 
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Ocean  Modeling  Using  High  Performance  Computing 

By  Dr.  Fred  T.  Tracy  and  Tommy  L  Biddlecome,  ERDC  MSRC,  ITL,  and  Dr  Wieslaw  Maslowski, 
Department  of  Oceanography  Naval  Postgraduate  School 


The  ability  to  predict  narrow  (of  order  10  km)  and  long 
(of  order  1,000  km)  features  in  sea  ice  is  of  great 
interest  and  importance  to  the  U.S.  Navy  submarine 
fleet  operations  in  the  region.  A  team  of  researchers 
from  the  Naval  Postgraduate  School  (NPS)  in 
Monterey,  California,  led  by  Dr.  Wieslaw  Maslowski, 
has  been  involved  in  the  development  of  the  Polar  Ice 
Prediction  System  (PIPS)  3.0  test  model.  This  is  a  part 
of  an  effort  sponsored  by  the  Office  of  Naval  Research 
to  upgrade  the  Navy  operational  PIPS  model.  The  final 
setup  for  ice/ocean  modeling  is  still  under  develop¬ 
ment.  The  main  emphasis  of  the  research  discussed  in 
this  article  is  on  improvements  of  the  modeled  sea  ice, 
leading  to  better  prediction  of  open- water  regions 
within  the  ice  pack  or  so-called  leads  and  polynyas 
frequently  observed  in  the  Arctic  Ocean. 

One  of  the  requirements  to  advance  model  skill  in 
predicting  leads  and  polynyas  involves  increasing 
model  horizontal  (and  vertical)  grid-cell  resolution. 

In  the  PIPS  3.0,  model  resolution  is  increased  to 
1/12-degree  (or  ~  9  km)  and  45  vertical  levels  from 


17  to  33  km  and  15  levels  in  PIPS  2.0  (see  Preller  et  al. 
2002,  for  more  details).  The  test  grid  uses  the  rotated 
coordinate  system  to  eliminate  a  singularity  problem  at 
the  North  Pole  and  to  minimize  the  latitudinal  grid 
convergence.  The  test  model  domain  (Figure  1)  in¬ 
cludes  the  North  Pacific  (from  about  30N),  the  proper 
Arctic  Ocean  with  the  Canadian  Archipelago  and  the 
Nordic  Seas,  and  the  North  Atlantic  (to  about  45N). 
This  approach  allows  the  test  model  to  account  for  all 
the  northern  latitude  ice-covered  regions  at  a  high  and 
nearly  uniform  resolution. 

Another,  more  logistical  requirement  is  related  to  a 
specific  architecture  target  for  running  the  PIPS  3.0  test 
model.  The  U.S.  Navy  Fleet  Numerical  Meteorology 
and  Oceanography  Center  (FNMOC)  has  committed  to 
the  modem  SGI  multiprocessor  technology,  such  as  the 
Origin  2000  (02K)  and  3000  (03K)  machines  in  their 
recent  operational  system  upgrades.  This  fact  imposed 
an  immediate  demand  for  changing  the  existing  NPS 
ocean  and  sea-ice  model  codes  (Maslowski  et  al.  2000, 
Maslowski  et  al.  2001)  from  using  SHMEM  on  Cray 


Naval  Postgraduate  School  -  Arctic  Modeling  Effort 


1/12  Degree  PIPS  3.0  Model  Bathymetry  (m) 


Figure  1.  Test  model  domain 
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T3E  machines  to  the  Message  Passing  Interface  (MPI) 
on  the  SGI  computers.  In  addition,  timely  information 
about  the  performance  of  the  MPI  version  of  the  PIPS 
3.0  test  code  on  the  03K  at  ERDC  was  critical  to 
determine  resources  needed  for  the  PIPS  3.0  on  the 
03K  computer  dedicated  to  run  the  Navy  operational 
codes  at  FNMOC. 

Support  of  Dr.  Maslowski  began  with  Dr.  Mark  Fahey, 
ERDC  MSRC  Computational  Science  and  Engineering 
(CS&E)  group  (now  at  Oak  Ridge  National  Labora¬ 
tory,  Oak  Ridge,  Tennessee).  Dr.  Fred  Tracy,  CS&E 
group,  and  Dr.  Fahey  ported  the  T3E  SHMEM  version 
of  PIPS  3.0  to  the  SGI  03K.  SHMEM  works  on  the 
03K,  but  adherence  to  the  rule  that  data  objects  being 
sent  and  received  must  be  symmetric  or  remotely 
accessible  is  critical.  A  symmetric  array  or  variable  has 
the  same  size,  type,  and  relative  address  on  all  process¬ 
ing  elements  (PEs).  One  can  consider  the  one-sided 
64-bit  get  shown  below. 

call  shmem_get64  (y,  x,  n,  ipe) 

where  n  pieces  of  64-bit  data  are  moved  from  x  to  y. 
Tracing  variables  such  as  x  and  y  back  through  the 
levels  (sometimes  several  subroutines)  of  the  various 
calls  was  done  to  ensure  that  when  they  were  originally 
defined,  they  were  in  a  common  block  rather  than  a 
dimension  statement.  Variables  in  a  common  block  are 
symmetric,  whereas  variables  defined  by  dimension 
statements  are  not.  With  a  liberal  use  of  barriers  for 
synchronization,  the  SHMEM  version  then  worked. 

A  joint  effort  between  Drs.  Maslowski  and  Tracy  has 
resulted  in  a  new  MPI-based  coupled  test  ice-ocean 
model.  To  do  this  quickly,  a  set  of  four  MPI  routines 
were  written  using  standard  MPI  (Version  I.l)  calls 
that  reproduced  the  capability  of  the  one-sided  commu¬ 
nication  SHMEM  gets  and  puts.  One-sided  communi¬ 
cation  is  written  in  the  MPI-2  standard  but  was  not 
implemented  on  many  architectures  at  that  time.  This 
approach  took  far  less  time  than  a  complete  rewrite  of 
the  message-passing  part  of  the  PIPS  3.0  test  code. 

This  new  code  has  been  evaluated  on  the  ERDC  03K 
complex  using  resources  of  the  NPS  Challenge  Project. 
Table  1  shows  a  comparison  of  PIPS  3.0  performance 
using  the  SHMEM  version  and  the  new  MPI  version 


Table  1.  PIPS  3.0  Performance  on  the  ERDC  03K 

Communication 

Number  of  PEs 

Time  (sec) 

SHMEM 

32 

1 ,484.71 

MPI 

32 

1,708.34 

SHMEM 

64 

991.92 

MPI 

64 

1,054.85 

for  a  test  data  set  on  the  ERDC  03K  complex  using 
32  and  64  PEs.  For  a  short-term  effort,  the  results  are 
good. 

The  test  model  results  show  a  dramatic  improvement  in 
the  modeling  of  sea  ice  and  ocean  behavior  in  response 
to  the  prescribed  atmospheric  forcing.  One  of  the  most 
challenging  aspects  of  modeling  the  Arctic  sea  ice  is  to 
properly  represent  its  thickness  distribution  throughout 
the  basin.  In  Figure  2,  the  3-year  mean  sea-ice  thickness 
distribution  for  1979-81  is  presented.  The  modeled 
distribution  of  ice  thickness  compares  reasonably  well 
with  that  known  from  observations.  Many  regions  are 
located  in  the  Arctic  Ocean  where  the  sea-ice  and 
ocean  observations  are  limited.  In  that  respect,  model 
results  are  potentially  useful  in  not  only  forecasting  ice 
thickness  and  ice-edge  variability  but  also  in  guiding 
future  field  campaigns  to  better  understand  environ¬ 
mental  changes  in  this  region. 

Case-specific  scientific  visualization  techniques  were 
developed  by  the  ERDC  Scientific  Visualization 
Center  (SVC)  staff  for  both  model  verification  and  data 
exploration  (Fang  et  al.  1998)  using  Dr.  Maslowski’s 
ocean  data.  Figures  3  and  4  show  example  results  of 
this  work.  Within  the  domain  of  large  data  sets  such  as 
this  test  model,  unique  difficulties  exist  in  exploration 
and  evaluation.  Therefore,  visualization  techniques 
were  used  to  represent  the  data  graphically  as  a  means 
of  gaining  understanding  and  insight  into  the  features 
within  the  data.  This  allows  the  researchers  to  more 
closely  examine  both  the  system  and  the  model  results, 
such  that  the  data  can  be  studied  in  ways  that  were 
previously  difficult  or  impossible.  The  coupled  arctic 
data  have  natural  physical  three-dimensional  represen¬ 
tations  that  can  be  usefully  explored.  The  visualization 
data  set  includes  45  parameters,  many  of  which  are 
best  represented  spatially.  These  include  such  variables 
as  sea-ice  thickness,  surface  snow  depth,  ice  concentra¬ 
tion,  flow  fields,  and  geographic  distributions  of 
changes  in  velocity. 

Data  set  size  can  become  a  computational  and  transfer 
bottleneck.  For  many  data  sets  or  simulations,  looking 
at  the  entire  data  set  in  detail  is  difficult.  Renderings 
for  this  project  were  created  for  365  time-steps  of 
45  variables,  where  each  variable  consisted  of  889,600 
values  resulting  in  ~1 17  GB  of  data  to  visualize. 

Using  an  indirect  geometry-based  technique,  structures 
in  scalar  or  vector  fields  were  visualized  by  converting 
them  first  into  surface  representations  and  then  render¬ 
ing  them  using  conventional  computer-graphics 
techniques.  This  means  produces  surfaces  in  the 
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Figure  2.  Three-year  mean  sea-ice-thickness  distribution  for  1979-81 


domain  of  the  scalar  quantity,  allowing  it  the  same 
value  as  the  surface  value.  Next,  the  surfaces  can  be 
colored  according  to  the  iso-surface  value  used  to 
generate  the  mesh.  Mapping  the  mesh  by  separate 
variables  also  gives  a  more  unique  coupled  view  into 
the  data. 

The  method  constructs  a  continuous  surface  from  the 
discrete  data  set,  transforming  data  into  graphical  data 
primitives  (in  this  case,  polygons).  Graphical  primi¬ 
tives  are  test  rendered,  and  the  geometrical  data  are 
converted  into  images.  Images  and  extracted  surfaces 
are  then  combined  to  produce  a  visualization.  Textures 
are  generated  to  color  surfaces,  e.g.,  those  generated  by 
the  geometric-extraction  techniques.  A  color  map  is 
used  to  assign  the  color.  At  the  same  time,  reference 
images  are  created  from  each  variable  in  each  time- 
step,  such  that,  for  any  specified  time,  a  color-mapped 
image  can  be  examined.  Movie  files  were  created  from 
a  sequence  of  rendered  images  of  each  of  the  variables 
showing  the  fluctuations  of  the  values  over  time. 

The  conceptual  animation  team  in  the  SVC  put  the 
scientific  data  into  context.  By  modeling  a  SeaWolf- 
class  submarine  and  rendering  animations  showing  the 
vehicle  emerging  through  a  low-density  section  of  the 
sea  ice,  the  animation  team  enabled  the  researcher  to 
view  the  scientific  data  in  a  useful  context.  An 
immersive  interactive  demonstration,  which  shows 


annual  variability  in  sea-ice  thickness  and  localized 
fluctuations  over  the  Arctic  Region,  was  created  for  the 
Immersadesk  for  displaying  the  coupled  ocean-climate 
general  circulation  model.  The  user  interactively 
explores  areas  of  interest  in  real  time. 

Also,  a  version  of  the  Parallel  Ocean  Program  (POP), 
version  2.0,  written  at  Los  Alamos  National  Labora¬ 
tory,  has  been  enabled  to  run  on  the  Cray  XI  for  this 
specific  application.  Drs.  John  Levesque  and  Alan 
Minga  of  Cray,  Inc.,  recently  provided  a  tuned  version 
of  POP  2.0  for  the  ERDC  Cray  XL  POP  2.0  does  high- 
resolution  global  and  regional  modeling. 
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Figure  3.  Ice  velocity 


Figure  4.  Latent  heat  flux 
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ERDC  MSRC  Computational  Science  and  Engineering  (CS&E) 
Group  Creates  a  New  Tool  for  Users 

By  Robert  M.  Hunter  and  Dr.  Ruth  Cheng 

A  majority  of  scientific  applications  require  a  computational  mesh,  which  is  a  discretized  form  of  the  physical 
domain.  The  mesh  is  represented  by  sets  of  vertices  and  edges,  which  form  elements  or  volumes.  A  variety  of  data 
can  be  associated  with  the  mesh,  and  these  data  can  sit  on  vertices  or  elements  of  the  mesh.  When  parallelism  is 
employed,  it  is  important  to  partition  the  mesh  evenly  among  processors,  while  maintaining  data  coherence  and  low 
communication  overhead.  This  can  be  hard.  DBuilder  has  been  developed  by  the  CS&E  staff  of  the  MSRC  to 
provide  a  simple  Application  Programming  Interface  (API)  for  users  to  sidestep  the  learning  curve  of  message 
passing,  graph  theorem,  and  parallel  algorithms.  DBuilder  implements  these  facets  of  parallel  programming  by 
utilizing  tools  and  libraries  such  as  ParMETIS  (Karypis  et  al.)  and  the  Message  Passing  Interface  (MPI). 

DBuilder  provides  support  of  domain  partitioning,  parallel  data  management,  coupling  coordination,  and  parallel 
solver  interface.  A  mesh  can  be  viewed  as  a  graph  G=(V,E),  which  is  a  set  of  vertices  (V)  and  edges  (E).  When  a 
subgraph  on  each  processor  is  given  to  DBuilder,  DBuilder  partitions  the  domain  (G)  to  processors.  This  partition 
may  be  manipulated  to  provide  a  distribution  that  is  low  in  communication  or  a  computationally  balanced 
workload.  DBuilder  can  build  a  vertex  domain  with  a  distributed  number  of  vertices,  an  element  domain  with  a 
distributed  number  of  elements,  or  a  boundary  element  domain  comprised  of  boundary  elements  in  the  element 
domain.  A  vertex  domain  is  favored  when  the  application  is  to  solve  vertex-based  values  such  as  the  finite  differ¬ 
ence  method,  while  an  element  domain  (or  the  vertex  domain  of  the  dual  graph  of  G)  is  for  solving  cell-centered 
values  such  as  the  finite  volume  method.  As  a  finite  element  application,  coordination  between  elements  and 
vertices  is  required  because  the  assembly  procedure  assembles  each  elemental  matrix  to  a  linear  system,  and  the 
linear  system  is  then  solved  for  vertex-based  values.  DBuilder  also  provides  a  default  rule  for  such  coordination 
between  vertices  and  elements.  A  callback  approach  is  supported  for  users  to  specify  their  own  rules. 

Data  migration  among  processors  is  encapsulated  in  DBuilder  using  MPI  data  types.  DBuilder  has  the  information  for 
local  and  ghost  vertices/elements  alive  throughout  the  entire  apphcation.  This  piece  of  information  is  mandatory  for 
gather/scatter  operation  in  the  parallel  paradigm.  Two  API  functions  are  facilitated  for  the  user  to  maintain  coherent  data 
structures  among  processors.  MPI  nonblocking  functions  are  called  in  DBuilder  for  data  synchronization. 

Eigure  1  shows  a  section  of  code  implemented  in  WASHI23D  (Cheng  et  al.)  to  illustrate  the  use  of  the  DBuilder 
API.  In  this  case,  both  vertex  and  element  domains  are  built  for  the  finite  element  method  to  maintain  a  balanced 
number  of  vertices  among  processors.  The  first  two  function  calls  pass  geometric  information  to  DBuild_Init 
for  building  vertex  and  element  domains,  respectively.  DBuilder  registers  their  memory  addresses  only  and  does  not 
keep  a  copy  of  the  geometric  information.  Then,  the  function  DBuild_Get_f  ile_part  can  be  called  to 
retrieve  the  local  count  of  data  to  be  read  on  the  processor.  Two  instances  of  DBuild_Domains  are  called  to 
build  the  vertex  domain  followed  by  the  element  domain.  The  element  domain’s  partition  rule  is  based  on  the 
constructed  vertex  domain.  These  two  domains  have  their  own  local  entities  and  ghost  entities,  where  the  ghost 
entities  are  owned  by  other  processors.  Two  steps  are  required  to  update  data  in  the  element  array  (i.e.,  element 
indices)  that  is  not  owned.  These  are  as  follows:  (I)  DBuild_Set_type  is  called  to  set  the  data  type  for  the 
element  array,  and  (2)  DBuild_Global_update  brings  in  the  ghost  values  to  the  element  array.  Based  on  the 
DBuilder  constructed  element  domain,  some  vertices  may  need  to  be  available  as  ghosts  on  a  given  processor, 
because  more  than  one  ghost  layer  may  be  necessary  in  the  algorithm.  DBuild_Add_ghos  t  s  is  available  to 
achieve  this  purpose.  DBuild_Set_type  and  DBuild_Global_update  are  then  called  to  bring  in  the 
vertex  coordinates  for  the  vertex  domain.  At  the  bottom,  DBuild_Set_type  is  called  to  set  up  the  data  type 
for  updating  the  data  on  ghost  vertices.  Obviously,  the  data  are  associated  with  vertices  because  the  data  type 
is  set  on  the  vertex  domain.  After  the  domains  are  built,  the  rest  of  parallelization  is  basically  to  place 
DBuild_Global_update  routines  at  appropriate  locations  in  the  code. 
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/***  Initialization  for  vtxDoamin  and  element Domain  ***/ 
ierr  =  DBuildInit {numglobalvertices ,  numghostlayer , 0 , 

point 2 ne ighbor_l i s t ,  pr oc_se t ,  Scvt x_ne i ghbor_l i s t , 

£cvtx_neighbor  list ,  &vtx_coordj  total  bytes  of  each  vtx, 
vtxDomain) ; 

ierr  =  DBuildInit (numglobalelements , numghostlayer , 
numneighbors  jerelm, 

&num_entries_per_elm, proc  set , &element_array , 
£celm_neighbor_list  ,NULL,  0,  element  Domain)  ; 

/***  Read  mesh  from  a  file  -  partial  file  is  read  on  each  processor  ***/ 

ierr  =  WashRead_geom3 (fd,  mesh);  /**  fill  in  coord  and  element  arrays 

in  mesh  and  then  create  neighbor  list  **/ 

/***  Build  domains  ***/ 

ierr  =  DBuild_Domains {1,  NULL,  vtxDomain); 

ierr  =  DBuildDomains { 3 ,  vtxDomain,  elementDomain) ; 

fiticit  update  element_array  for  ghost  elements  ***/ 
ierr  =  DBuildSettype (numentriesperelm* sizeof (int ) , &dType , 
elementDomain) ; 

ierr  =  DBuildGlobalupdate (elementarray, dType , elementDomain) ; 

/***  add  ghost  vertices  based  on  elementDomain  ***/ 
ierr  =  DBuild_Add_ghosts (element_array, num_local_vertices, 

num  neighbors  per  elm, num  entries  per  elm, vtxDomain) ; 

/***  bring  in  the  coordinates  for  ghost  vertices  ***/ 

ierr  =  DBuildSettype (num_dir*sizeof (double) , &dType, vtxDomain) ; 

ierr  =  DBuildGlobalupdate {coord, dType , vtxDomain) ; 

/***  build  data  types  for  data  gathering  or  scattering  ***/ 

ierr  =  DBuildSettype (si zeof (double) , feme sh->doubleType, vtxDomain) ; 


Figure  1.  Code  containing  DBuilder  APIs 

Multiphysics  applications  on  multidomains  have  become  a  large  focus.  These  applications  require  that 
multidomain  integration  be  executed  to  integrate  two  or  more  applications.  The  spatial  relationship  between 
computational  domains  can  be  adjacent,  partially/fully  overlapped,  or  distinct.  DBuilder  allows  for  the  building  of  a 
coupler  object  to  avoid  the  dependency  between  meshes  when  partitioning.  Figure  2  depicts  the  concept  of  this 
coupler  implementation.  In  the  figure,  all  four  processors  participate  in  two-dimensional  (2-D)  and  3-D  simula¬ 
tions.  Because  dependency  is  not  specified  between  domains  when  partitioning,  a  given  processor  in  the  3-D 
domain  exclusively  owning  a  2-D  slice  of  the  domain  is  not  guaranteed.  As  shown  in  the  figure,  processor  P3  owns 
a  partition  of  the  2-D  subdomain,  which  is  a  partition  of  the  3-D  subdomain  owned  by  processor  P2  and  P3. 
Likewise,  the  top  of  the  3-D  subdomain  that  processor  PI  owns  is  a  partial  subdomain  partitioned  to  PO  and  PI  in 
2-D.  A  coupler  formed  with  DBuilder  will  provide  the  message  passing  from  2-  to  3-D  and  from  3-  to  2-D  as 
shown  in  the  red  and  blue  dashed  arrow  lines,  respectively. 
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Figure  3  shows  a  section  of  code  used  to  build  a  vertex 
coupler  associated  with  the  vertex  domain  and  an 
element  coupler  associated  with  the  element  domain. 

The  API  function  DBuild_Coupler_init  is 
called  twice  to  create  these  two  couplers.  The  function 
DBuild_Get_coupler_size  is  called  for  a 
request  of  the  size  of  a  coupler.  The  argument 
DB_D2  TODl  accounts  for  the  size  of  the  coupler  in 
Domain  2  (D2)  when  updating  Domain  1(D1).  From 
the  code  in  Figure  3,  DBuilder  internally  represents  the 
the  2-D  domain  as  D1  and  the  3-D  domain  as  D2.  This 
is  determined  by  the  ordering  of  the  domains  in  the  call 
to  DBuild_Coupler_init.  The  function 
DBuild_Coupler_update  is  called  to  maintain 
consistent  data  on  two  different  domains  (e.g.,  D2  and 
Dl)  among  processors.  In  this  example,  the  data  on  D1 
named  mtyp_temp,  which  is  a  source,  update  the 
destination  on  D2  shown  in  the  first  argument  based  on 
the  coupler’s  element  domain  mentioned  at  the  last 
argument.  According  to  the  third  argument,  each  entry 
being  composed  of  two  integers  is  known. 

Figure  2.  Concept  of  the  coupler  implementation 


/***  initialization  and  creation  of  coupler's  vtx  domain  and  element 
domain  **+/ 

ierr  =  DBuild_Coupler_init  (&mesh2 - >vtxDomain,  S:mesh3 - >vtxDomain, 
coupler- >npxz ,  mesh2 - >vtxDomain , numberLo cal Elements , 
&coupler->vtx_coupler) ; 

ierr  =  DBuild_Coupler_init (&mesh2 - >element Domain, &mesh3 - >elementDomain, 
coupler- >mtopxz , mesh2 - >e lament Domain . number Local Elements , 
&coupler->elm_coupler) ; 

/ic*^  get  the  size  of  coupler's  element  domain  ***/ 

ierr  =  DBuildGetcouplersize { &couplerSize , ^coupler- >elm_coupler , 
DB_D2TODl) ; 

/***  update  Dl's  vector  to  D2 '  s  vector  on  the  element  domain 

ierr  =  DBuildCouplerupdate (coupler- >mtypxz ,  mtyptemp,  2*sizeof (int) , 
DB_D1T0D2, ^coupler- >elm_coupler) ; 


Figure  3.  Code  building  coupler  using  DBuilder 

DBuilder  has  an  interface  to  linear  solvers  such  as  BlockSolve95  (Jones  and  Plasserman)  and  pARMS  (Yousef  et 
al.).  As  shown  in  Figure  4,  only  two  functions  are  required  to  use  the  parallel  linear  solver  BlockSolve95.  First, 
DBuild_Solver_reset_co  passes  the  matrix  and  associated  geometric  domain  to  DBuilder.  Second, 
DBuild_Solver_solve  along  with  the  right-hand  side  vector,  initial  guess,  and  the  associated  mesh  domain  is 
passed  to  DBuilder  to  solve  the  linear  system  with  the  result  stored  in  the  second  argument  and  the  residual  infor¬ 
mation  at  the  third  argument. 


ierr  =  DEuild_Solver_reset_co  (matrix,  &mesh3->vtxDomaiii)  ; 

iter  =  DBuild_Solver_solve (rhs,  x,  ^residual,  &mesh3“>vtxDomain) ; 

Figure  4.  Code  calling  D Builder's  API  to  interface  BlockSolve95 
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In  summary,  the  full  functionality  of  a  parallel  software  toolkit  is  implemented  in  DBuilder.  Experimental  results 
(Figure  5)  show  that  the  overhead  for  building  domains  can  be  considered  insignificant.  However,  the  time  spent  in 
communication  to  maintain  the  data  coherence  becomes  dominant  as  the  number  of  processors  increases.  Reasons 
for  a  communication-bound  application  can  be  that  the  problem  size  is  too  small  for  a  large  number  of  processors 
and  the  parallel  solver  is  not  efficient,  i.e.,  too  much  communication  time  spent  in  the  solver.  Thus,  more  sophisti¬ 
cated  parallel  solvers,  such  as  BlockSolve95  and  pARMS,  are  integrated  into  DBuilder.  Figure  6  shows  the  perfor¬ 
mance  improvement  using  BlockSolve95.  Currently,  DBuilder  is  installed  in  the  /usr/local/usp  directories 
on  the  HPC  machines,  and  a  users  manual  will  be  available  shortly. 


Performance  Measurement 
alpha_test_3d 


—•—Time  in  Building 
Domains 

—■—Time  in  Global 
Updates 

—A— Total  Execution 
Time 

- Ideal  Speed  Up 


number  of  processors 


Figure  5.  Performance 
measurement  on 
Compaq  SC45 


Figure  6.  Performance  comparison  between  two 
iinear  solvers  on  Compaq  SC45 
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Development  of  a  Comprehensive  Groundwater  Model  of  the  Defense 
Distribution  Depot  San  Joaquin  (DDJC)y  Sharpe  Site,  Lathrop,  California 

By  Barbara  R  Donnell,  ERDC  CHL,  David  R.  Richards,  ERDC ITL,  Earl  V.  Edris,  ERDC  CHL, 

Dr.  Thomas  L  McGehee,  ERDC  CHL,  and  Dr  Fred  T  Tracy,  ERDC  MSRC,  ITL 


The  Defense  Distribution  Depot  San  Joaquin-Sharpe, 
in  Lathrop,  California  (DDJC-Sharpe),  is  an  active 
military  installation  acting  as  a  supply  depot  for  the 
Defense  Logistics  Agency  (see  Figure  1  for  a  site 
map).  It  has  been  active  for  many  years,  and  in  the 
course  of  its  activities,  solvents  used  in  degreasing 
operations  have  contaminated  the  groundwater.  The 
primary  contaminant  of  concern  is  trichloroethylene 
(TCE),  which  presents  itself  in  three  distinct  plumes  at 
DDJC-Sharpe  (Figure  2).  To  address  the  contamination 
problem,  the  installation  operates  three  pump-and-treat 
remediation  systems  to  contain  the  three  plumes  and  to 
treat  the  contaminated  groundwater. 

Until  the  initiation  of  this  effort,  a  comprehensive 
groundwater  model  had  not  been  developed  at  DDJC- 
Sharpe  that  was  designed  specifically  to  provide 
accurate  predictions  of  flow  and  transport  over  a  wide 
range  of  hydrologic  conditions.  These  include  such 
varied  conditions  as  the  siting  of  new  municipal  water 
wells  nearby,  variations  in  the  surface-water  drainage 
and  stormwater  retention  ponds,  and  optimization  of 
the  pump-and-treat  systems  to  get  better  contaminant 
capture  for  less  cost. 

ERDC  was  asked  by  the  Huntsville  Engineering  and 
Support  Center  to  collaborate  in  the  development  of 
such  a  groundwater  model  for  DDJC-Sharpe  (Donnell 
et  al.  2003).  The  purpose  of  the  modeling  was  to 
manage  the  remediation  activities  at  DDJC-Sharpe  and 
to  determine  the  ultimate  fate  of  contaminants  given  a 
variety  of  proposed  hydrologic  conditions. 

Using  numerical  models  in  planning  and  conducting 
remediation  operations  at  Department  of  Defense 
(DoD)  installations  has  become  commonplace.  Indeed, 
a  substantial  investment  has  been  made  by  DoD  in  the 
development  of  a  state-of-the-art  modeling  system  to 
assist  in  these  cleanup  efforts.  The  reasons  are  clear 
since  contamination  at  DoD  installations  is  significant, 
cleanup  costs  are  great,  and  the  overall  DoD  budget  is 
decreasing.  Modeling  systems  are  essential  for  the 
DoD  to  optimize  remediation  and  provide  cost- 
efficient  cleanup. 

While  numerical  models  have  a  broad  range  of  uses, 
narrowly  focusing  the  purpose  of  the  modeling  exer¬ 
cise  is  important.  One  modeling  approach  is  not 
appropriate  for  all  sites,  just  as  treatment  technologies 


Regional  Map  for  DDJC-Sharpe  and  DDJCTracy  Installation  Locations 


Figure  1.  Site  map  of  the  region  of  interest 

for  contamination  are  site  specific.  Modeling  does  not 
solve  all  problems  at  a  cleanup  site  but  does  indeed 
improve  the  chance  for  remediation  success. 

The  specific  purpose  of  this  modeling  exercise  is  to 
develop  a  regional  scale  model  that  can  address  a  wide 
range  of  groundwater  management  questions.  The 
developed  model  will  be  housed  within  a  graphically 
driven  user  environment  that  will  allow  the  installation 
to  add  data  as  they  are  collected  and  to  build  a  “living” 
groundwater  model  of  the  installation.  The  system  will 
be  “living”  in  that  it  will  exist  as  an  integral  part  of  the 
remediation  management  plan  throughout  its  life. 

The  model  will  continue  to  improve  through  time  as 
additional  data  or  model  improvements  are  added. 

With  these  additions,  the  model  accuracy  will  improve 
as  will  its  adaptability;  and,  therefore,  remediation 
effectiveness  will  be  easier  to  demonstrate.  Ultimately, 
the  performance  of  the  remediation  action  and  the 
model’s  ability  to  forecast  this  behavior  will  be  crucial 
in  managing  the  treatment  practices  over  the  remaining 
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Figure  2.  North  Balloon,  Central  area,  and  South  Balloon  historical  TCE  concentration  contours  at  the 
DDJC-Sharpe  site.  Concentration  data  and  figure  provided  by  URS  Corporation,  the  onsite  contractor 


life  of  the  remediation  project.  This  includes  the 
predictions  to  questions  of  when  to  end  pumping  and 
what  potential  benefits  will  be  realized  from  natural 
attenuation  of  the  contaminants. 

The  modeling  system  chosen  for  the  DDJC-Sharpe 
application  is  the  DoD  Groundwater  Modeling  System 
(GMS)  (Groundwater  Modeling  Team  2004).  GMS 
was  designed  and  constructed  precisely  for  the  type  of 
application  presented  at  DDJC-Sharpe.  GMS  contains 
a  wide  variety  of  database,  conceptualization,  visual¬ 
ization,  simulation,  and  optimization  tools  to  develop 
remediation  designs  and  manage  remediation  efforts. 

Unstructured  finite  element  models  such  as 
FEMWATER  (Lin  et  al.  1997)  are  more  useful  in 
complicated  geometric  settings.  Numerically,  a  need 
for  mesh  resolution  flexibility  exists  at  DDJC-Sharpe 
since  the  location  and  number  of  wells  or  surface- 
drainage  features  could  be  changed  dramatically  from 
one  simulation  to  the  next.  Since  many  wells  can  be 
simulated,  with  each  well  providing  a  potentially  steep 
cone  of  depression,  a  need  to  apply  higher  grid  resolu¬ 
tion  only  where  the  gradients  occur  exists.  FEMWATER 
also  has  the  advantage  of  variably  saturated,  coupled 
flow  and  transport. 


Eor  these  reasons,  the  numerical  model  selected  for 
application  at  DDJC-Sharpe  was  EEMWATER. 
EEMWATER  was  originally  put  into  the  GMS  with 
support  provided  by  the  U.S.  Environmental  Protection 
Agency  (USEPA),  Region  IV.  The  reason  for  its 
inclusion  was  to  address  complexities  in  groundwater 
flow  problems  not  suitably  addressed  in  other  models. 

EEMWATER  has  been  converted  to  the  parallel 
computational  environment  (Tracy  2000).  This  provided 
the  production  tool  to  accurately  model  complex 
unsaturated  groundwater  problems  such  as  the  percola¬ 
tion  pond  scenarios.  In  order  to  conduct  large-resolution 
simulations,  a  parallel  computational  environment, 
such  as  the  ERDC  Cray  T3E,  was  required.This 
technology  provides  the  capability  to  divide  a  large 
problem  into  a  discrete  number  of  smaller  problems 
that  can  be  simultaneously  solved  on  a  set  of  proces¬ 
sors.  Quality  control  measures  were  conducted  to 
ensure  comparability  between  the  serial  and  the 
parallel  versions  of  EEMWATER.  On  average,  ap¬ 
proximately  22  minutes  was  the  run  time  for  64 
parallel  processors  to  solve  the  steady-state  flow 
simulations  needed  as  compared  with  13  hours  on  an 
available  serial  workstation. 
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A  transport  simulation  using  the  parallel  version  of 
^  FEMWATER  is  now  being  conducted  for  this  project 
(Donnell  2004).  Preliminary  50-year  transport  simula- 
tions  with  pumping  are  taking  37  hours  on  average  of 
run  time  using  128  parallel  processors  on  the  ERDC 
I  CrayT3E. 
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Cray  XI  Impact 

By  Dr.  Paul  M.  Bennett,  Robert  Alter,  and  Dr  Fred  Tracy 


Cray  X1  ”Bring-Your-Own-Code’* 
Workshop 

The  ERDC  MSRC  hosted  a  "Bring- Your-Own-Code" 
workshop  for  users  new  to  the  Cray  XI  architecture  on 
March  2-4,  2004.  The  Cray  XI,  a  liquid-cooled,  multi¬ 
streaming  vector  supercomputer,  was  delivered  to  the 
ERDC  MSRC  in  June  2003  and  became  fully  available 
to  users  on  January  1,  2004.  Its  peak  computational 
performance  of  800  billion  floating-point  operatons  per 
second  (Gflop/s)  is  provided  by  64  multistreaming 
processors,  each  containing  four  single- streaming 
processors  and  4  GB  of  RAM,  for  a  total  of  256  GB  of 
memory.  Topics  discussed  at  the  Cray  XI  workshop, 
held  in  the  ERDC  Information  Technology  Laboratory 
Collaboratorium,  included  hardware  design,  techniques 
for  porting  and  optimizing  codes,  parallel-ization 
techniques,  compiler  features,  tools  for  debugging  and 
performance  analysis  of  codes,  and  a  Portable  Batch 
System  (PBS)  for  submitting  jobs  and  determining  job 
status.  The  workshop  was  organized  and  led  by  Dr. 

Paul  Bennet  of  the  ERDC  MSRC.  Dr.  John  Levesque 
of  Cray  described  XI  architecture  and  programming 
techniques.  Dr.  Ered  Tracy  of  the  ERDC  MSRC  gave  a  presentation  on  graph  coloring  strategies  that  allow 
multistreaming,  and  Dr.  Andrew  Johnson  of  the  Army  High  Performance  Computing  Research  Center  (AHPCRC) 
gave  a  presentation  on  his  CED  code.  Participants  included  representatives  from  the  major  services  and  agencies 
that  compute  in  the  High  Performance  Computing  Modernization  Program  (HPCMP)  from  sites  across  the  country. 
The  primary  focus  of  the  workshop  was  to  assist  users  in  immediately  migrating  their  codes  to  the  XI  and  then 
tuning  them  for  better  perforamnce.  Members  of  the  ERDC  MSRCs  Computational  Science  and  Engineering 
(CS&E)  group  were  paired  with  user  attendees  to  accomplish  this,  and  significant  progress  was  made  on  most  of 
the  codes.  The  workshop  was  so  well  received  that  the  ERDC  MSRC  is  planning  future  workshops  to  further 
facilitate  the  XI  code  migration  process. 


Dr  John  Levesque,  a  Cray  Applications  Speciaiist, 
served  as  an  instructor  at  the  ERDC  MSRC’ s 
Cray  X1  ‘Bring-Your-Own-Code”  workshop 


Dr  Steve  Ketcham,  far  right,  of  the  ERDC  CRREL,  received  exceiient  resuits  with 
PTOP3D  at  the  ERDC  MSRC’s  Cray  X1  ‘Bring-Your-Own-Code”  workshop 
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More  Cray  X1  Impact 


Dieter  Postl,  a  doctoral  candidate  working  with 
Dr.  Herman  Fasel  of  the  Department  of  Aerospace  and 
Mechanical  Engineering  at  the  University  of  Arizona,  is 
using  NST3DD  to  perform  computational  fluid  dynamics 
(CFD)  simulations  of  low-pressure  turbulent  flows  for 
various  generic  geometries.  NST3DD  solves  the  time- 
dependent  incompressible  Navier-Stokes  equations  using 
a  pseudo-spectral  method  with  Fourier  components  in  the 
z-direction  and  compact  4th-order  finite  differences  in  the 
X-  and  y-directions.  The  code  was  originally  parallelized 
across  the  Fourier  modes  using  OpenMP.  However,  the 
high-memory  bandwidth  causes  scahng  to  rapidly 
deteriorate  for  more  than  four  CPUs.  The  code  was 


Recent  computations  requiring 
roughly  82  million  grid  points 
on  10  nodes  of  the  Cray  XI  have 
had  turnaround  time  less  than 
24  hours,  compared  with  longer  than 
3  months  on  an  SGI  Origin  3400. 


migrated  to  a  vectorized  multi-streamed  MPI  code  on  the 
Cray  XI,  whose  large  intemode  and  intranode  memory 
bandwidth  allows  much  more  rapid  processing.  Recent 
computations  requiring  roughly  82  million  grid  points  on 
10  nodes  of  the  Cray  XI  have  had  turnaround  time  less 
than  24  hours,  compared  with  longer  than  3  months  on  an 
SGI  Origin  3400. 


_ HPC  at  Work 

New  Data  Translator  Enables  Visualization  of  CTH  Results  with 
the  General  Mesh  Viewer 

By  Dr.  Richard  A.  Weed 


The  CTH  hydrocode  developed  at  Sandia  National 
Laboratory  is  one  of  the  standard  tools  used  by 
Department  of  Defense  (DoD)  researchers  for  explo¬ 
sive  effects  analysis  and  force  protection  design.  An 
essential  part  of  any  CTH  analysis  is  the  visualization 
of  results  at  various  times  in  the  simulation  cycle  to 
determine  the  current  status  of  simulation.  In  addition, 
visualization  tools  that  can  display  all  or  part  of  a 
simulation  interactively  in  three  dimensions  are 
required  for  determining  where  problems  might  be 
occurring  in  the  simulation.  However,  the  visualization 
tools  supplied  with  CTH  are  based  on  two-dimensional 
graphics  packages  that  do  not  allow  users  to  interact 
with  the  data  in  three  dimensions.  CTH  users  at 
several  DoD  sites  in  discussions  with  members  of  the 
HPCMP  Programming  Environment  and  Training 
(PET)  program  Computational  Structural  Mechanics 
(CSM)  team  identified  enabling  visualization  of  CTH 
data  with  widely  available  three-dimensional  (3-D) 
visualization  tools.  Two  of  these  tools  are  TECPLOT 
and  the  General  Mesh  Viewer  (GMV)  program  from 
Los  Alamos  Laboratories  as  a  potential  PET  core 
support  activity.  Based  on  these  requests  by  DoD 
users,  the  development  of  a  series  of  data  translators 
that  can  convert  CTH  data  into  formats  readable  by 
several  different  visualization  tools  was  undertaken  by 
PET. 

The  decision  was  made  that  the  first  of  these  translators 
would  be  targeted  at  generating  data  that  could  be  viewed 
with  the  GMV  program  since  GMV  is  freely  available, 
noncommercial  software  that  runs  on  a  wide  variety  of 
UNIX  workstations  and  HPC  systems.  Therefore,  it  can 
be  installed  on  individual  user  workstations  without  the 
expense  of  a  software  hcense.  In  addition,  GMV  is 
currently  used  by  researchers  at  ERDC  to  visualize 
results  from  another  hydrocode  (SHAMRC).  GMV  is  an 
OPENGL  application  with  a  wide  variety  of  features  for 
viewing  data  generated  by  Einite  Element  and  Einite 
Volume  codes  interactively  in  three  dimensions.  These 
features  include  support  for  most  of  the  commonly  used 
finite  element  shapes,  cut  planes,  rendering  of  iso¬ 
surfaces  and  iso-volumes,  and  viewing  multiple  material 
data.  In  addition,  a  parallel  version  of  GMV  exists  for 
batch  rendering  of  large  data  sets.  Since  the  GMV  viewer 
can  distinguish  between  the  internal  binary  data  formats 
of  different  systems,  GMV  data  files  can  be  created  on 
any  HPC  system  and  viewed  on  user  local  workstations 
without  external  translation  of  the  data  file  into  the 


internal  binary  format  of  the  workstation. 

Development  of  a  CTH  to  GMV  translator  was  begun 
in  2003.  After  reviewing  the  EORTRAN  77  code  used 
by  CTH  to  create  restart  and  plot  data  files,  the  deci¬ 
sion  was  made  that  the  basic  routines  required  to  read 
CTH  data  files  would  be  rewritten  in  EORTRAN  95  to 
enhance  code  portability  and  long-term  maintainability. 
This  approach  also  eliminates  the  need  to  have  access 
to  the  CTH  code  base  and  libraries  for  users  who 
would  like  to  build  their  own  versions  of  the  data 
translators.  In  addition,  a  library  of  C  utilities  provided 
by  Los  Alamos  for  creating  GMV  files  was  also 
implemented  in  EORTRAN  95  along  with  a  small 
library  of  C  utilities  of  EORTRAN-compatible  wrap¬ 
pers  around  C  intrinsic  functions  required  to  create  C 
stream  data  format  files  used  by  GMV.  The  resulting 
code  base  for  the  translator  is  predominately  EOR¬ 
TRAN  95  code  with  only  a  small  percentage  of  the 
required  code  written  in  C.  This  eliminates  many  of  the 
problems  associated  with  migrating  mixed  EORTRAN 
and  C  applications  across  different  hardware  platforms. 
In  addition,  most,  if  not  all,  of  the  nonstandard  con¬ 
forming  code  used  in  the  CTH  EORTRAN  77  routines 
such  as  dynamic  memory  allocation  using  CRAY 
pointers  can  be  replaced  by  EORTRAN  95  code  that 
rigidly  adheres  to  the  EORTAN  95  standard.  The  plan 
is  that  all  of  the  C  code  currently  used  to  access  the 
underlying  C  EO  utilities  will  be  replaced  with  logic 
that  conforms  to  the  C  interoperability  features  pro¬ 
posed  for  the  forthcoming  EORTRAN  2003  standard 
that  is  scheduled  for  final  approval  in  2004. 

A  beta  version  of  the  CTH  to  GMV  data  translator 
named  CTH2GMV  was  released  for  public  review  in 
January  of  2004.  The  beta  version  is  a  scalar  applica¬ 
tion  that  supports  reading  of  both  single-processor  and 
parallel  CTH  data  files.  Users  can  select  specific  CTH 
variables  to  dump  at  user- specified  time-steps.  Most  of 
the  variables  that  can  be  plotted  with  the  existing  CTH 
visualization  programs  can  be  extracted  by  CTH2GMV. 
User  input  for  CTH2GMV  is  based  on  a  simple 
keyword-oriented  format  that  duplicates  many  of  the 
keywords  used  for  existing  CTH  inputs.  Much  effort 
was  devoted  to  keeping  the  required  input  to  a  mini¬ 
mum.  The  following  example  illustrates  all  of  the  data 
required  to  extract  cell  pressure  for  all  of  the  CTH 
time-steps  stored  in  multiple  CTH  plot  files  (plcth) 
generated  by  a  parallel  run  and  create  a  GMV  file  with 
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a  root  file  name  of  runl_p  for  each  time- step  that 
contains  the  requested  data. 

CTH2GMV  input  example: 

$/usr/local/usp/CTH02/CTH2GMV/bin/cth2gmv 
CTH2GMV  >  mf=plcth  gf=run1 
CTH2GMV  >  get  all  var=p 
CTH2GMV  >  quit 

CTH2GMV  also  supports  writing  multiple  variables  to 
each  GMV  input  file.  In  addition,  three  data  decima¬ 
tion  procedures  can  be  selected  to  reduce  the  size  of 
the  output  data  and  restrict  the  region  of  the  CTH  grid 
from  which  data  are  extracted.  CTH2GMV  can  read 
both  CTH  restart  and  plot  (plcth)  files. 


Development  of  CTH2GMV  is  still  ongoing  with  final 
release  planned  later  in  2004.  A  parallel  MPI-based 
version  of  the  code  is  planned  for  the  final  release 
along  with  support  for  simulations  that  used  Adaptive 
Mesh  Refinement.  In  addition,  future  translators  are 
planned  to  support  TECPLOT  and  the  S ANDIA 
EXODUS  database  format. 

The  beta-release  version  can  be  accessed  by  members 
of  the  CTH  group  on  the  ERDC  COMPAQ  SC40/45 
and  SGI  ORIGIN  systems.  ERDC  MSRC  CTH  users 
interested  in  trying  out  CTH2GMV  should  contact 
Dr.  Richard  Weed  (RichardA.Weed@erdc.usace.army. 
mil),  ERDC  PET  Onsite  Lead  for  CSM,  for  more 
details  on  accessing  and  running  the  code. 


The  following  figures  illustrate  CTH  pressures  on  the  symmetry  planes  and  interior  cut  planes  generated  by  the 

detonation  of  an  explosively  formed  projectile 


The  next  figure  illustrates  volume  fraction  contours  for  the  explosively  formed  projectile  simulation 
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technology  update 


EasyHPC  Debuts  in  Beta  Test 


High  performance  computing  is  on  its  way  to  becom¬ 
ing  easier,  thanks  to  a  new  Web-based  utility  developed 
by  the  Software  Engineering  and  Evaluation  Branch  of 
the  Engineering  and  Informatics  Systems  Division  in 
ITL  in  partnership  with  the  ERDC  MSRC.  EasyHPC 
allows  users  to  upload/download  files,  manage  and 
submit  batch  jobs,  cancel  jobs,  display  job  statistics 
and  CPU  usage,  and  provide  command  line  execution 


to  the  ERDC  MSRC  parallel  computers  from  the 
comfort  and  familiarity  of  their  normal  desktop 
environment.  The  EasyHPC  Beta,  targeted  initially  at 
Windows  users,  entered  beta  evaluation  with  a  small 
group  of  HPC  users  on  April  13,  2004.  The  Beta 
Review  Team  is  advising  the  development  group  on 
final  tweaks  to  the  software  before  general  distribution, 
expected  later  this  fiscal  year. 


. tips  and  techniques 

Simple  Techniques  for  Improving  Performance  -  Case  Study 


By  Bobby  Hunter 

The  Deformation  Mapping  Technique  (DMT),  a  serial 
code  that  identifies  deformations  induced  from  stresses 
applied  to  a  specimen,  has  been  recently  improved. 

The  technique  compares  pixels  in  a  digital  image  taken 
before  the  straining  event  with  a  digital  image  taken 
afterwards.  The  technique  knows  no  scale  boundaries 
and  can  be  applied  to  deformation  at  the  microscopic 
level  as  well  as  at  the  macroscopic  level.  This  enables 
deformation  imaging  of  a  range  of  scales,  from  large 
structures  such  as  buildings  and  bridges  to  micrometer¬ 
sized  points  in  an  experimental  specimen.  Total 
runtime  on  example  executions  on  the  ERDC  MSRC’s 
Origin  3900  was  reduced  from  37,000  to  51  seconds, 
an  improvement  of  725-fold.  This  parallelization  and 
optimization  process  permits  more  images  to  be 
analyzed  in  a  given  time  frame  at  a  reduced  cost  to 
process  the  data  and  enables  researchers  to  analyze 
material  deformation  to  a  greater  depth.  As  a  result, 

10  image  pairs  can  now  be  analyzed  in  1  day  compared 
with  the  previous  metric  of  one  pair  in  2  days. 


As  a  result,  10  image  pairs  can  now  be  analyzed 
in  1  day  compared  with  the  previous  metric 
of  one  pair  in  2  days 


Reducing  wall  clock  time  from  37,000  to  -7,000 
seconds,  without  code  modifications,  was  done  by 
using  compiler  optimizations.  The  code  was  compiled 
with  -03  flag  passed  to  the  compiler,  which  turns  on 
aggressive  optimization.  Floating  point  accuracy  may 
be  affected  as  the  compiler  rearranges  floating-point 
instructions,  resulting  in  rounding  variances  from  the 
nonoptimized  code.  Also,  the  compile  time  will 
increase  as  the  compiler  generates  high-quality  code. 
To  ensure  that  the  -03  code  generated  acceptable 
results,  the  output  from  the  code  built  with  optimiza¬ 


tions  was  validated  against  output  results  from 
nonoptimized  code.  For  this  particular  case,  the  03 
results  were  deemed  acceptable  as  long  as  the  maxi¬ 
mum  difference  between  the  03  results  and  OO  results 
was  no  greater  than  lOe-6. 

DMT  was  an  exceptional  candidate  for  parallelization, 
as  it  was  embarrassingly  parallel.  By  this,  it  means  that 
the  data  could  be  divided  into  independent  “subdata” 
blocks  and  the  same  computation  performed  for  each 
subdata  block  on  an  independent  processor.  Therefore, 
a  processor  could  work  on  a  given  portion  of  the  grid 
without  a  need  for  interaction  with  other  processors. 
The  only  parallel  logic  needed  was  to  determine  on 
which  points  of  the  grid  each  processor  would  work. 
For  this  particular  problem,  the  grid  was  divided  into 
column  strips  for  distribution  across  processors.  The 
following  code  snippet  demonstrates  the  algorithm 
executed  by  each  processor  for  determining  the  strips 
that  it  would  compute. 

MPI_lnit(^argc,  ^argv); 

MPI_Comm_size(M  PI_COMM_WORLD,  number_processors); 
M  P I  _C  om  m_ran  k  (M  P  l_COM  M_WORL  D .  my_p  rocess_id ) ; 
remainder  =  total_points_x  %  number_processors: 
my_numberjDoints_x  =  total  jDoints_x  /  number j^rocessors; 
if  (my_process_id  <  remainder) 
m  y_n  u  m  be  r_poi  nts+ + ; 

my_startjpoint_x  =  my_number_points_x  *  my^processjd; 
if  (my_process_id  >=  remainder) 

my_startjpoint_x  +=  remainder; 

Only  four  calls  to  the  Message  Passing  Interface  (MPI) 
were  essential  in  parallelizing  the  code;  three  of  these 
are  included  in  the  above  code: 

MPIJnitO 

MPLComm_size(); 

M  P I  _C  om  m_ran  k() ; 

MP(_Finalize(): 

MPI-IO  routines  were  used  for  parallel  output  of  the 
solution  but  were  not  a  necessity.  For  more  information 
on  MPI  routines,  visit  http://www.mpi-forum.org. 
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The  ERDC  MSRC  Challenge  Projects  Support  Team 

By  Dr.  Paul  M.  Bennett 


Each  year,  the  Deputy  Under  Secretary  of  Defense  for 
Science  and  Technology,  together  with  the  Director  of 
the  DoD  High  Performance  Computing  Modernization 
Program  (HPCMP),  selects  Challenge  Projects  for 
allocations  on  the  total  HPC  resources  provided  by  the 
HPCMP.  Each  Challenge  Project  may  run  for  up  to  3 
years,  with  possible  extension  for  another  6  months. 

The  ERDC  MSRC  has  been  assigned  19  of  these 
Challenge  Projects  for  FY2004.  These  Challenge 
Projects  account  for  roughly  10  percent  of  the  total 
hours  available  at  the  ERDC  MSRC.  The  projects  are 
as  follows: 

>  Directed  High  Power  RE  Energy:  Foundation  of 
Next-Generation  Air  Force  Weapons 

>  High  Fidelity  Analysis  of  UAVs  Using  Non-Linear 
Fluid  Structure  Simulation 

>  High  Fidelity  Simulation  of  Littoral  Environments 

>  Seismic  Signature  Simulations  for  Tactical  Ground 
Sensor  Systems 

>  Multiscale  Simulations  of  Nanotubes  and  Quantum 
Structures 

>  Multiscale  Simulations  of  High  Energy  Density 
Material 

>  Evaluation  and  Retrofit  for  Blast  Protection  in 
Urban  Terrain 


>  First  Principal  Studies  of  Technologically  Impor¬ 
tant  Smart  Materials 

>  Numerical  Modeling  of  Turbulent  Waves  for  Naval 
Applications 

>  3-D  CFD  Modeling  of  the  Chemical  Oxygen- 
Iodine  Laser  II 

>  Simulation  of  Coherent  Radar  Backscatter  from 
Dynamic  Sea  Surfaces 

>  3-D  Modeling  and  Simulation  of  Weapons  Effects 
for  Obstacle  Clearance 

>  Computational  Support  for  Chemically  Reactive 
Flows  and  Non-Ideal  Explosives 

>  Stochastic  Simulations  of  Flow  Structure 
Interactions 

>  Towards  Predicting  Scenarios  of  Environmental 
Arctic  Change 

>  Scalable  Simulations  of  Combustion  Chamber 
Dynamics  and  Hypergolic  Gel  Propellant 
Chemistry  for  Selectable  Thrust  Engines  in 
Next-Generation  Guided  Missiles 

>  Tip-to-Tail  Turbulent  Scramjet  Flowpath 
Simulation  with  MHD  Energy  Bypass 

>  High  Accuracy  DNS  and  LES  of  High 
Reynolds  Number,  Supersonic  Base  Flows, 
and  Passive  Control  of  the  Near  Wake 


ERDC  MSRC  Challenge  Projects  POCs:  (seated  left  to  right)  Dr  Paul  Bennett,  Chris  Fuson, 

Bobby  Hunter,  Carrie  Leach,  Dr  Nathan  Prewitt,  and  Bob  Aiter;  (standing  left  to  right)  Tommy  Biddiecome, 
Paul  Adams,  Mike  Gough,  Richard  Walters,  Dr  Sam  Cable,  and  Dr  Fred  Tracy  (Not  pictured  are  Phii  Bucci, 
Dr.  Jeff  Hensley,  Dr.  Tom  Oppe,  Bill  Renaud,  and  David  Sanders) 
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The  ERDC  MSRC  assists  its  Challenge  Projects 
with  a  team  of  Points-of-Contact  (POCs)  com¬ 
prised  of  Government  and  contractor  employees. 
The  Challenge  Support  Team  includes  members 
of  the  Computational  Science  and  Engineering 
group,  the  Scientific  Visualization  Center,  the 
Programming  Environment  and  Training  (PET) 
program  team,  and  the  Customer  Assistance 
Center.  Dr.  Bennett  is  the  ERDC  MSRC  Chal¬ 
lenge  Projects  Coordinator.  The  Challenge 
Projects  Support  team  monitors  the  progress  that 
the  Challenge  Projects  make  against  their  alloca¬ 
tions  and  works  closely  with  the  ERDC  MSRC 
Customer  Assistance  Center  to  resolve  any  issues 
that  may  impede  performance  of  the  Challenge 
computational  tasks.  The  team  also  assists  with 
code  migration  and  tuning. 

Communication  plays  a  vital  role  in  resolving 
issues  and  monitoring  the  Challenge  Projects. 
The  Challenge  Projects  Coordinator  regularly 
reviews  system  issues  and  the  status  of  each 
Challenge  Project.  The  POCs  meet  regularly  to 
discuss  the  status  of  their  respective  Challenge 
Projects  and  any  issues  or  problems  that  the 
Projects  may  have.  Additionally,  the  Coordinator 
meets  with  each  POC  as  needed  to  discuss  and 
resolve  issues  requiring  greater  attention,  such  as 
software  purchases,  allocation  reimbursements, 
or  hardware  problems.  This  approach  allows  the 
Challenge  Projects  Support  Team  to  support  the 
ERDC  MSRC  Challenge  Projects  with  a  high 
level  of  success. 

For  more  information  regarding  the  Challenge 
Projects  at  the  ERDC  MSRC,  please  contact 
Dr.  Paul  Bennett  through  the  Customer  Assis¬ 
tance  Center  at  msrchelp®  erdc.hpc.mil  or 
telephone  1-800-500-4722. 


Honing  the  Edge 

By  Gerald  R.  Morris 


Rikk  Anderson  discusses  pipeline  hazard 
mitigation  techniques 


As  part  of  the  continuing  effort  to  provide  professional 
development  opportunities  to  its  staff,  the  ERDC 
MSRC  is  undertaking  a  seminar  series  on  computer 
architecture.  The  intent  is  to  give  ERDC  MSRC  staff 
members  detailed  insight  into  the  inner  workings  of 
modern  digital  computers.  The  seminars,  which  are 
being  facilitated  by  Dr.  Bill  Ward,  ERDC  MSRC 
Computational  Science  and  Engineering  lead,  began  in 
January  2004  and  will  continue  through  December 
2004.  Each  participating  member  is  responsible  for 
preparing  and  presenting  a  seminar  on  some  computer 
architecture  topic.  Topics  span  the  entire  range  of 
computer  architecture,  including  fundamentals  of 
computer  design,  instruction  set  architectures, 
pipelining,  memory  hierarchy,  multiprocessors,  storage 
systems,  and  interconnection  networks.  The  textbook 
used  for  the  seminar  series  is  Hennessy  and  Patterson's 
classic  text.  Computer  Architecture:  A  Quantitative 
Approach,  3rd  Edition. 
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Supercomputing  2003  *^Igniting  Innovations^’ 


By  Rose  J.  Dykes 

The  Phoenix  Civic  Plaza  Convention  Center,  Phoenix, 
Arizona,  was  the  site  of  the  15th  supercomputing 
conference.  Highlighting  the  most  innovative  develop¬ 
ments  in  high  performance  computing  and  networking, 
SC2003  brought  together  researchers,  engineers, 
scientists,  educators,  etc.,  presenting  projects  that  they 
had  in  the  works  and  computer  hardware  and  software 
vendors  showing  off  their  wares. 

TeraFLOPS -level  machines  presented  at  this  confer¬ 
ence  have  come  a  long  way  from  when  peak  perfor¬ 
mance  was  an  “impressive”  3  gigaFLOPS  at  the  first 
SC  conference  held  in  Orlando,  Florida,  in  1988.  This 
conference  has  highlighted  important  innovations  of 
the  information  age  starting  with  the  birth  and  growth 
of  the  World  Wide  Web. 

Dr.  Ruth  Cheng  represented  the  ERDC  MSRC  at  the 
conference  technical  program  with  her  presentation  of 
a  poster  entitled  “Parallel  Implementation  of  the 
WASH123D  Code  for  Surface  and  Subsurface  Hydro- 
logic  Interactions.”  Paul  Adams  and  Richard  Walters 
presented  scientific  visualization  lectures  as  well  as 
demonstrations.  Robert  Athow  and  Dr.  Wayne  Mastin 
explained  the  PET  program  to  booth  visitors. 

The  NAVO  MSRC  was  tasked  with  the  responsibility 
for  the  DoD  HPCMP  booth  for  SC2003.  Work  has 
already  begun  at  ERDC  for  the  SC2004  Conference  to 
be  held  in  Pittsburg,  Pennsylvania,  November  6-12, 
2004.  The  ERDC  MSRC  will  be  heading  up  the  DoD 
HPCMP  booth  this  fall,  supporting  the  SC2004 
“Bridging  Communities”  theme. 


Dean  Hampton  (center),  ERDC  MSRC,  and 
Dr.  George  Heburn  (far  right),  Mississippi 
State  University,  answer  questions 
at  the  DoD  HPCMP  booth 


Dr.  Ruth  Cheng  at  the  Conference  Poster  Session 


Professor  Shahrouz  Aiiabadi  (left),  Clark  Atlanta 
University,  and  Robert  Athow,  ERDC  MSRC, 
at  the  HPC  in  Mississippi  booth 
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HPC  Community  .  .  . 

Department  of  Energy  Computer  Graphics  Forum 

Paul  Adams,  ERDC  MSRC  Scientific  Visualization  Lead,  made  a  presentation  entitled  “Large-Scale  Visualizations 
of  Department  of  Defense  Challenge  Projects”  at  the  Department  of  Energy  Computer  Graphics  Lorum  in  Santa  Le, 
New  Mexico,  on  April  27,  2004.  He  discussed  the  process  by  which  visualization  of  these  large-scale  simulations  is 
performed.  Over  60  people  attended  the  forum  from  various  national  laboratories  including  those  from  Los  Alamos, 
Lawrence  Livermore,  and  Sandia,  and  the  High  Performance  Computing  Modernization  Program  (HPCMP) 
centers.  The  goal  of  scientific  visualization  is  enabling  the  DoD  researchers  to  discover  phenomena  occurring  in  the 
simulation  that  they  had  previously  not  been  able  to  observe.  The  expertise  of  the  Scientific  Visualization  team  is 
available  to  users  of  the  HPCMP  computing  resources. 

ERDC  MSRC  Video  Winning  Entry  at  American  Physical  Society  Meeting 

An  ERDC  MSRC  user,  Antonino  Lerrante,  in  conjunction  with  Paul  Adams,  Miguel  Valenciano,  and  David 
Longmire  of  the  Scientific  Visualization  Center  presented  a  video  at  the  56th  American  Physical  Society  (APS) 
meeting.  Division  of  Lluid  Dynamics.  The  video  was  among  the  four  winners  of  the  Gallery  of  Lluid  Motion  videos 
at  the  APS-DLD  2003.  The  video  was  entitled  “Evolution  of  Quasi-Stream  wise  Vortex  Tubes  and  Wall  Streaks  in  a 
Bubble-Laden  Turbulent  Boundary  Layer  over  a  Plat  Plate.”  The  visualization  showed  resulting  vortex  tubes  from 
the  direct  numerical  simulation  of  turbulent  flow  over  a  flat  plate.  The  video  was  shown  at  the  American  Physical 
Society  meeting  in  March,  and  a  short  article  will  appear  in  the  Physics  of  Pluids  journal  in  September  2004.  It  will 
then  be  placed  on  the  Web  site  of  the  APS  Gallery  of  Lluid  Motion  (http://www.aps.org/units/dfd/index.html). 

^MaxweWs  Equations”  Paper  Published  in  Elsevier  Journal  Physica  D 

Dr.  Paul  Bennett,  ERDC  MSRC,  and  Dr.  Alejandro  Aceves,  University  of  New  Mexico  Department  of  Mathematics 
and  Statistics,  published  the  paper  “Parallel  Numerical  Integration  of  Maxwell’s  Pull- Vector  Equations  in  Nonlinear 
Pocusing  Media”  in  Physica  D,  Vol.  184,  pp  352-375,  October  1,  2003.  This  paper  presents  results  of  parallel 
numerical  integration  of  Maxwell’s  equations  in  bulk  Kerr  glass.  The  index  of  refraction  of  Kerr  materials  depends 
upon  the  intensity  of  the  coherent  light.  In  this  study,  ultra-short  optical  pulses  of  light  were  propagated  through  a 
regular  rectangular  grid  in  order  to  study  the  distance  to  blowup  of  sufficiently  intense  pulses. 

IEEE  Visualization  2003  Conference 

Paul  Adams,  ERDC  MSRC  Scientific  Visualization  lead,  presented  a  paper  at  the  IEEE  Visualization  2003  Confer¬ 
ence  in  Seattle,  Washington,  on  October  23,  2003.  The  paper  was  entitled  “Visualization  of  Steep  Breaking  Waves 
and  Thin  Spray  Sheets  Around  a  Ship.”  The  Simulation  and  Visualization  of  the  Breaking  of  Waves,  the  Formation 
of  Thin  Spray  Sheets,  and  the  Entrainment  of  Air  Around  the  Next  Generation  of  Naval  Surface  Combatants  is  an 
ongoing  3-year  DoD  Challenge  project.  The  goal  of  this  project  is  to  develop  a  validated  computation  capability  to 
model  the  full  hydrodynamics  around  a  surface  combatant,  including  all  of  the  processes  that  affect  mission  and 
performance.  The  novel  visualization  of  these  large-scale  simulations  enabled  the  researchers  to  discover  phenom¬ 
ena  occurring  in  the  simulation  that  they  had  previously  not  been  able  to  see. 

SIAM  Conference  on  Parallel  Processing  for  Scientific  Computing 
Participation 

Drs.  William  A.  Ward,  Jr.,  and  Ruth  Cheng,  both  with  the  ERDC  MSRC,  made  presentations  at  the  Society  for 
Industrial  and  Applied  Mathematics  (SIAM)  Conference  on  Parallel  Processing  for  Scientific  Computing.  The 
conference  was  held  February  25-27  in  San  Francisco,  California.  Dr.  Cheng  presented  a  paper  entitled  “Develop¬ 
ment  of  Parallel  Particle  Tracking  Algorithms  in  Large-Scale  Unsteady  Flows.”  Her  presentation  focused  on  the 
development  of  an  Ordinary  Differential  Equation  solver  to  solve  trajectory  equations  in  unsteady  flows.  The 
application  of  the  overland  flow  problems  solved  by  the  diffusion  wave  model  was  demonstrated. 

Dr.  Ward’s  presentation,  “Experiences  Profiling  and  Characterizing  DoD  High  Performance  Computing  Applica¬ 
tions,”  defined  the  goals  of  gathering  profile  data  for  the  HPCMP  profiling  project,  identified  the  three  levels  of 
profiling  to  be  used  to  characterize  the  computational  behavior  of  DoD  HPC  applications,  and  described  the  bench¬ 
mark  application  codes  analyzed  by  the  MSRC  for  the  HPCMP  office. 
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A  Joint  Outreach  Effort  of  the  ERDC  and  NAVO  MSRCs  Expands 
Awareness  of  High  Performance  Computing  in  the  State  of 
Mississippi 

By  Rose  J.  Dykes 

ERDC  MSRC  Director  John  E.  West  and  NAVO 
MSRC  Director  Stephen  A.  Adamec  spoke  at  the 
inaugural  session  of  the  2003-04  Evening  Seminar 
Lecture  Series  on  High  Performance  Computing,  one 
part  of  an  ongoing  effort  between  the  Department  of 
Defense  and  various  universities  in  Mississippi.  The 
primary  goal  of  the  lecture  series  was  to  increase 
awareness  of  HPC  within  the  State  of  Mississippi, 
through  focused  efforts  among  DoD,  academia,  and 
private  industry. 

West  and  Adamec  presented  talks  at  the  University  of 
Southern  Mississippi  (USM)  on  October  23  that 
complemented  one  another,  with  Adamec  presenting 
perspectives  on  present  and  future  HPC  capabilities 
and  West  focusing  on  HPC  applications  of  the  research 
challenges  facing  supercomputing  in  the  future.  The 
session,  geared  toward  preparing  the  next  generation  of 
science,  engineering,  and  HPC  workers,  was  attended 


by  more  than  150  students  and  faculty.  Extensive 
question  and  answer  periods  were  held  after  both  talks. 

On  February  26,  Paul  Adams,  ERDC  MSRC  Scientific 
Visualization  Center,  and  Pete  Gruzinskas,  NAVO 
MSRC  Scientific  Visualization  Center,  both  spoke  on 
HPC  for  scientific  visualization  at  USM  and  then  again 
on  the  same  topic  at  Jackson  State  University  (JSU)  on 
March  11. 

West  spoke  again  at  JSU  on  March  4  about  the  user 
perspective  on  HPC  and  how  to  prepare  for  careers  in 
HPC.  On  March  25,  Jay  Clibum,  ERDC  MSRC 
Technical  Operations  Manager,  ended  the  lecture  series 
with  a  presentation  entitled  “High  Performance 
Computing:  An  Operations  Manager’s  Perspective”  at 
USM.  His  presentation  included  an  overview  of 
parallel  computing  and  various  HPC  architectures. 


Vicksburg  High  School  Students  Shadow  ERDC  MSRC  Team  Members 

Students  from  Vicksburg  High  School,  Vicksburg,  Mississippi,  recently  visited  ERDC  to  shadow  ERDC  team 
members  that  work  in  fields  in  which  they  are  interested.  Dr.  Bill  Ward  and  Paul  Adams  both  of  the  MSRC  partici¬ 
pated  in  this  program.  Dr.  Ward  is  the  Computational  Science  and  Engineering  lead,  and  Paul  Adams  is  the  Scien¬ 
tific  Visualization  Center  lead. 


ERDC  Team  Members  Judge  a  Local  Elementary  School  Reading  Fair 


Ginny  Miller,  ERDC  MSRC,  Julia  Baker,  ERDC  GSL, 
and  Maggie  Sabol,  ERDC  CHL,  participated  on 
January  13,  2004,  as  Reading  Fair  judges  at  Sherman 
Avenue  Elementary  School  in  Vicksburg,  Mississippi. 
After  reviewing  125  entries  from  grades  1-4,  judges 
selected  20  winners,  including  five  first-place  finishers. 
Judges  rated  the  projects  based  on  clarity  of  writing, 
creativity,  quality,  thoroughness,  and  interest  evoked. 
Winners  will  go  on  to  compete  in  Reading  Fairs  at  the 
district,  regional,  and  state  levels. 
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During  Career  Day  activities,  local  high  school  stu¬ 
dents  learned  that  the  ERDC  MSRC  is  about  more 
than  supercomputers.  The  high  performance  computing 
(HPC)  center  is  home  to  a  variety  of  careers,  from 
scientific  visualization  and  customer  assistance  to 
technical  writing  and  Web  design. 

“There  are  many  career  opportunities  in  the  field  of 
high  performance  computing,”  said  Jeanie  McDonald, 
who  organized  the  ERDC  MSRC’s  participation  in  the 
Career  Day  events  at  St.  Aloysius  High  School  in 
Eebruary  and  at  Vicksburg  High  School  and  Warren 
Central  High  School  in  October. 


Shaping  the  future 

High  School  Students  Learn  About  the  ERDC  MSRC  During 
Career  Day  Activities 

By  Ginny  Miller 


Technology  Transfer  Specialist  Rose  Dykes  speaks  to 
students  during  Career  Day  activities  at  St  Aloysius 
High  School 


During  the  2003-04  Career 
Day  activities,  ERDC 
MSRC  employees  not  only 
spoke  to  students  in  grades 
9-12  about  career  opportu¬ 
nities  but  also  encouraged 
them  to  attend  college  and 
to  make  smart  decisions. 

The  ERDC  MSRC  also 
participates  in  job 
shadowing  programs, 
tours,  and  internships. 


Video  Production  Specialist  David  Longmire  speaks  to  St  Aloysius  High  School 

students  during  Career  Day 


Career  Day  participation  is  just  one  education  initiative 
at  the  ERDC  MSRC,  whose  leadership  realizes  the 
responsibility  of  training  tomorrow’s  workforce.  “One 
of  our  missions  is  to  attract  our  best  and  brightest 
students  to  science  and  engineering,”  ERDC  MSRC 
Director  John  West  said. 

“Retaining  them,”  he 
added,  “will  ensure 
technological  leadership 
for  the  next  generation.” 
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Avis  Phillips  speaks  to  Warren  Central 
High  School  students  during 
Career  Day  activities 


Jeanie  McDonald  addresses  students  at 
Warren  Central  High  School  during 
Career  Day  activities 


Richard  Walters  of  the  ERDC  MSRC  Scientific  Visualization 
Center  demonstrates  the  SVC’s  capabilities  to  a  group  of 
Vicksburg  High  School  students 
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announcements 


New  Commander  of  ERDC  Visits  the  MSRC 

By  Rose  J.  Dykes 


David  Stinson,  ERDC  MSRC,  expiains  the  relationship  of 
the  ERDC  MSRC  to  the  DoD  High  Performance  Computing 
Modernization  Program  to  COL  Rowan 
as  Drs.  Dent  and  Robert  Maier, 

ERDC  MSRC,  stand  by 


ITL  Deputy  Director,  Dr.  Deborah  Dent  (far  left),  and 
ITL  Director,  Dr.  Jeffery  Holland  (far  right),  have  just 
presented  an  ITL  overview  to  COL  James  Rowan 
(center)  before  he  tours  the  ERDC  MSRC  HPC  and 
Scientific  Visualization  centers 

said.  All  of  COL  Rowan’s  previous  assignments  have 
been  with  active  duty  military  units,  and  he  has  been 
an  engineer  officer  for  25  years. 

A  West  Point  graduate,  COL  Rowan  earned  two 
master’s  degrees — one  in  Operations  Research  and 
Systems  Analysis  from  the  Georgia  Institute  of  Tech¬ 
nology  and  one  in  Military  Arts  and  Science  from  the 
Army  Command  and  General  Staff  College.  He  also 
received  a  degree  from  the  U.S.  Army  War  College. 


Paul  Adams  (second  from  left).  Scientific  Visualization 
lead,  explains  to  COL  Rowan  how  the  data  from  the  high 
performance  computers  are  visualized  in  a  context  that  will 
help  to  ensure  a  better  understanding  of  a  project.  Looking 
on  in  the  background  (left  to  right)  are  Tommy  Biddlecome 
and  Richard  Walters,  Scientific  Visualization  Center, 
and  David  Stinson 


COL  James  R.  Rowan  visited  the  MSRC  on  March  18. 
His  interest  in  high  performance  computing  (HPC)  and 
scientific  visualization  was  apparent  as  he  asked 
questions  and  commented  on  the  state-of-the-art 
infrastructure  and  technology  that  he  observed. 


“Ensuring  that  the  U.S.  warfighter  has  the  advantage 
on  the  battlefield  with  the  help  of  high  performance 
computing  is  very  important  to  me,  the  Corps  of 
Engineers,  the  Army,  and  the  DoD.  The  ERDC  MSRC 
plays  a  major  role  in  helping  provide  this  advantage,” 
said  COL  Rowan. 


Immediately  after  COL  Rowan  assumed  command  of 
ERDC  on  July  26,  2003,  he  was  deployed  to  Iraq  on  a 
6-month  assignment  as  the  deputy  theater  engineer  for 
Combined  Joint  Task  Eorce  Seven.  He  officially 
assumed  command  of  ERDC  on  March  10,  2004. 


“I  am  thrilled  to  have  the  opportunity  to  serve  as  the 
ERDC  commander.  This  assignment  is  different  from 
any  other  I’ve  had  to  date.  I  requested  this  assign¬ 
ment.  .  .1  have  been  an  ERDC  customer  for  many  years, 
reaching  back  for  solutions  on  airfield  construction, 
well-drilling,  bridging,  snow  and  ice  clearing,  river 
modeling,  and  communications  from  Germany,  Bosnia, 
Albania,  and,  most  recently,  Iraq.  This  is  my  first 
opportunity  to  serve  with  the  Corps  and  my  first 
chance  to  be  part  of  a  research  organization,”  Rowan 
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RSS:  A  New  Way  of  Communicating  with  the  ERDC  MSRC 

By  Ginny  Miller 


The  ERDC  MSRC  launched  an  ambitious  project  in 
January  to  improve  the  way  it  communicates  with 
users.  Really  Simple  Syndication,  or  RSS,  is  a  method 
of  distributing  headlines  and  other  Web  content  in  an 
XML  format.  The  ERDC  MSRC  is  the  first  in  the 
HPCMP  to  implement  the  user-friendly  technology. 

“We’re  doing  this  for  our  users,”  said  John  West, 

ERDC  MSRC  Director.  “This  is  a  new  format  for 
reaching  out  and  communicating  with  people.” 

Users  may  subscribe  to  any  or  all  of  the  ERDC  MSRC 
channels  by  supplying  the  URLs  of  the  RSS  newsfeeds 
they  want  to  receive  through  an  RSS  reader.  When  a 
channel  has  been  added,  the  reader  then  does  all  the 
work,  searching  for  chosen  RSS  sites  -  at  a  frequency 
determined  by  the  user  -  and  automatically  alerting 
users  when  new  content  is  available. 

“This  is  a  simple  way  to  push  information  to  the  users 
on  their  own  terms,”  West  said.  Prior  to  RSS,  the 
ERDC  MSRC  Customer  Assistance  Center  notified  its 
nearly  1,000  users  of  system  outages,  platform  avail¬ 


ability,  holiday  schedules,  and  other  ERDC  MSRC 
events  exclusively  through  e-mail.  Believing  the 
barrage  of  “user  messages”  could  be  replaced  with 
something  better.  West  suggested  RSS  as  a  new 
method  of  supplying  data  to  users. 

The  project  was  first  tested  by  staff  last  fall  and  made 
available  to  users  at  the  beginning  of  the  year. 

The  ERDC  MSRC  currently  provides  five  RSS 
newsfeeds  for  users.  The  General  Interest  channel 
conveys  a  broad  range  of  news  and  announcements 
about  the  Center,  while  four  system- specific  channels 
provide  updates  about  system  problems  and  events 
affecting  the  Cray  XI,  Cray  T3E,  the  SGI  Origins,  and 
Compaq  SCs.  News  is  posted  as  it  happens,  from  once 
a  week  to  several  times  a  day. 

Eor  more  information  about  RSS,  visit  the  ERDC 
MSRC  Web  site  at  www.erdc.hpc.mif  or  contact  the 
ERDC  MSRC  Customer  Assistance  Center  at  1-800- 
500-4722  or  msrchelp@erdc.hpc.mil. 


Psst! 


WeVe  got  a  better  way 


to  communicate. 
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National  Geographic  Film  Crew  Shoots  Footage  at  the  ERDC  MSRC 

By  Rose  J.  Dykes 


The  production  company  producing  a  show  for  Na¬ 
tional  Geographic  on  the  Murrah  Building  bombing  in 
Oklahoma  City  filmed  at  ERDC  on  January  30.  This 
show  that  will  be  Part  3  of  a  12-Part  series  on  forensic 
engineering.  Because  of  the  computing  done  here,  the 
MSRC  was  one  of  the  sites  chosen  for  filming  at 
ERDC. 

Dr.  Paul  Mlakar,  ERDC  Geotechnical  and  Structures 
Laboratory  (GSL),  was  the  on-camera  spokesperson 
for  the  ERDC  filming.  The  advanced  research  capabili¬ 
ties  and  force  protection  technologies  of  the  Corps  of 
Engineers  was  demonstrated  with  the  filming  of 
animations  of  high  performance  computing  simulations 
relating  to  blasts  and  structures.  This  was  filmed  in  the 
MSRC  Scientific  Visualization  Center.  Eootage  was 
shot  in  the  MSRC  High  Performance  Computing 
Center  as  Dr.  Mlakar  discussed  the  capabilities  of  the 
supercomputers. 

Part  3  of  the  12-Part  series  of  the  documentary,  the 
filming  at  ERDC,  will  air  in  late  2004  or  early  2005. 


Dr.  Mlaker  waits  for  the  camera  to  roll  before  he 
discusses  the  visualization  on  the  Immersadesk 


National  Geographic  filming  crew  arrives  at  the 
Information  Technology  Laboratory 


(Left  to  right)  Wayne  Stroupe,  ERDC  Public  Affairs 
Office,  quizzes  two  of  the  crew  members  about  their 
equipment  as  Dr  Paul  Mlakar,  ERDC  GSL,  and  John 
E.  West,  ERDC  MSRC  Director,  watch  outside  the 
ERDC  MSRC  Scientific  Visualization  Center  (SVC) 


(Left  to  right)  John  West,  Wayne  Stroupe,  and  the 
film  crew  producer  discuss  the  footage  to  be  filmed 
in  the  HPC  Center 


John  West  talks  about  the  fine  points  of  the 
visualizations  with  Dr  Mlaker  as  the  crew  sets 
up  in  the  SVC 
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ERDC  MSRC  Conducts  a  New  User  Workshop 


The  ERDC  MSRC  hosted  a  new  user  workshop  orga¬ 
nized  by  Dean  Hampton  for  new  users  on  April  20, 2004. 
Bill  Renaud  presented  an  overview  of  the  ERDC  HPC 
machines.  He  discussed  other  topics  such  as  obtaining 
accounts,  the  Load  Sharing  Eacility  (LSE)  scheduler 
system,  disk  space  management,  using  authentication 
software,  how  to  write  and  submit  scripts  to  run  jobs, 
using  the  mass  storage  system,  and  the  utilities  available. 


Bill  Renaud  making  his  presentations 


New  users  (seated)  and  Dr.  Ruth  Cheng  and  Bobby 
Hunter  (standing  left  to  right),  ERDC  MSRC 


Bobby  Hunter  and  Dr.  Ruth  Cheng  packaged  the 
parallel  WASH123D  software,  demonstrated  parallel 
job  submission/launches,  and  showed  the  EasyHPC 
system  (see  page  17).  The  purpose  of  this  workshop 
was  to  aid  in  the  migration  to  ERDC  HPC  machines, 
promote  the  parallel  WASH123D  code  to  a  production 
stage,  and  support  users  for  the  Philadelphia  District’s 
technical  project. 


ERDC  Researcher  Named  to  the  User  Advocacy  Group 

Dr.  Stephen  Ketcham,  ERDC  Cold  Regions  Research  and  Engineering  Laboratory,  was  named  in  March  as  a 
member  of  the  User  Advocacy  Group  (UAG)  for  users  of  the  DoD  HPCMP.  The  group’s  purpose  is  to  influence 
policies  and  practices  of  the  program,  to  facilitate  the  exchange  of  information  between  the  user  community  and 
the  HPCMP,  to  serve  as  an  advocacy  group  for  all  HPCMP  users,  and  to  advise  the  DoD  HPCMP  Office  on  policy 
and  operational  matters  related  to  the  HPCMP.  Membership  consists  of  four  members  from  each  Service  and  one 
member  representing  participating  DoD  agencies.  As  a  member  of  the  UAG,  Dr.  Ketcham  will  ensure  that  the 
needs  of  ERDC  users  of  the  HPCMP  resources  will  be  identified  and  met.  (Taken  from  the  April  30th  ERDC 
Rollup) 


/ - \ 

The  ERDC  MSRC  welcomes  comments  and  suggestions  regarding  The  Resource  and  invites 
article  submissions.  Please  send  submissions  to  the  following  e-mail  address: 

msrchelp@erdc.hpc.mil 

\ _ _ _ / 
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visitors 


Edward  J.  Welch  (front,  second  from  left),  Chief  of  Police,  New  York  City  Police 
Department,  with  Paul  Franco,  U.S.  Army  Engineer  District,  New  York  (front,  far  left), 
Marc  Ryan,  General  Dynamics  Corportation  (behind  Edward  Welch),  Dr.  Jeffery  Holland 
(to  right  of  Edward  Welch),  ERDC  ITL  Director,  Dr.  Bob  Welch  (second  from  right), 
ERDC  ITL,  and  John  E.  West  (far  right),  ERDC  MSRC  Director,  April  22,  2004 


(Front  row)  The  Honorable  John  Paul  Woodley,  Jr.,  Assistant  Secretary  of  the  Army, 
Civil  Works  (center),  with  Dr.  Jeffery  Holland  (far  left)  and  John  West  (far  right). 
(Back  row,  left  to  right)  COL  James  R.  Rowan,  ERDC  Commander,  Dr.  Dave  Pittman, 
GSL  Acting  Director,  and  Dr.  James  Houston,  ERDC  Director,  April  22,  2004 
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visitors 


John  West  (foreground)  talks  with  (left  to  right)  Dr.  Jimmy  Johnston,  U.S.  Geological 
Survey;  Dr.  John  H.  Marburger,  Science  Advisor  to  President  George  W.  Bush,  and 
serves  as  Director  of  the  White  House  Office  of  Science  and  Technology  Policy; 
Dr.  Houston;  COL  Rowan;  and  Dr.  Gene  Whitney,  National  Science  and  Technology 
Council.  Dr.  Charles  G.  Groat  (back,  center).  Director,  U.S.  Geological  Survey 
was  also  with  the  group,  March  24,  2004 


(Left  to  right)  Don  Wilson,  Navigation  Branch  Chief,  ERDC  CHL,  and  from  the  Portland 
District,  Bob  Buchholz,  Hydraulic  Design  Section  Chief;  Ron  Mason,  Hydraulics  and 
Hydrology  Branch  Chief;  Mike  Ott  and  Laurie  Rice,  Engineers  in  Training,  Engineering 
and  Construction  Division;  and  LTC  Charles  Markham,  Deputy  District  Engineer/Acting 

District  Engineer,  March  1 7,  2004 
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visitors 


David  Stinson  (far  right),  ERDC  MSRC, 
with  Mid- West  Natural  Resources 
Group  Spring  Meeting  attendees, 
March  2,  2004 


(Left  to  right)  Mr.  Andy  Bruzewicz,  Office 
of  Homeland  Security;  Ed  Hecker,  Chief, 
Office  of  Homeland  Security;  Dr.  Reed 
Mosher,  ERDC  GSL;  John  Meador, 
Deputy  Chief,  Infrastructure  Support 
Branch,  Office  of  Homeland  Security; 
Dr.  Cary  Butler,  ITL  Technical  Director; 
Dr.  Bob  Welch,  ITL;  and  Bob  Athow, 
ERDC  MSRC,  January  22,  2004 


(Standing)  John  West  talks  with  Dr.  Dixon 
Butler,  Committee  on  Appropriations, 
U.S.  House  of  Representatives  (seated 
at  table);  Drs.  Holland  and  Houston 
(seated  behind  Dr.  Butler);  and  Miguel 
Valenciano,  ERDC  MSRC  Scientific 
Visualization  Center,  January  6,  2004 
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acronyms 


Below  is  a  list  of  acronyms  commonly  used  among  the  DoD  HPC  community.  You  will  find  these  acronyms 
throughout  the  articles  in  this  newsletter. 


AHPCRC 

Army  High  Performance  Computing 
Research  Center 

API 

Application  Programming  Interface 

BB 

Big  Brother 

CFD 

Computational  Fluid  Dynamics 

CHL 

Coastal  and  Hydraulics  Laboratory 

CPU 

Central  Processing  Unit 

CRREL 

Cold  Regions  Research  and  Engineering 
Laboratory 

CSM 

Computational  Structural  Mechanics 

DMT 

Deformation  Mapping  Technique 

DNS 

Direct  Numerical  Simulation 

DoD 

Department  of  Defense 

DOE 

Department  of  Entergy 

ERDC 

Engineer  Research  and  Development  Center 

FNMOC 

Fleet  Numerical  Meteorology  and 
Oceanography  Center 

GMS 

Groundwater  Modeling  System 

GMV 

General  Mesh  Viewer 

GSL 

Geotechnical  and  Structures  Laboratory 

HPC 

High  Performance  Computing 

HPCMP 

High  Performance  Computing 

Modernization  Program 

ITL 

Information  Technology  Laboratory 

LES 

Large-Eddy  Simulation 

LSF 

Load  Sharing  Facility 

MPI 

Message  Passing  Interface 

MSP 

Multistreaming  Processor 

MSRC 

Major  Shared  Resource  Center 

NJDEP 

New  Jersey  Department  of  Environmental 
Protection 

NPS 

Naval  Postgraduate  School 

PBS 

Portable  Batch  System 

PE 

Processing  Element 

PET 

Programming  Environment  and  Training 

PIPS 

Polar  Ice  Prediction  System 

RSS 

Really  Simple  Syndication 

SIAM 

Society  for  Industrial  and  Applied 
Mathematics 

SMS 

Surface-Water  Modeling  System 

SSP 

Single- Streaming  Processor 

SVC 

Scientific  Visualization  Center 

TCE 

Trichloroethylene 

UAG 

User  Advocacy  Group 

USEPA 

U.S.  Environmental  Protection  Agency 

training  schedule 


For  the  latest  on  PET  training  and  on-line  registration,  please 
go  to  the  On-line  Knowledge  Center  Web  site: 

https  ://okc.  erdc.  hpc.  mil 

Questions  and  comments  may  be  directed  to  PET  training 
at  (601)  634-3131,  (601)  634-4024,  or 
PET-  Training  @  erdc.  usace.  army,  mil 
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Psst! 


We’ve  got  a  better  way 


to  communicate . 


Web  based  service. 

You  decide  what  news  to  receive. 
Call  us..  ,  we’ll  show  you  howl 


Here’s  why  you’ll  get  pOW6rful  results 
with  RSS (Really  Simple  Syndication): 

•  Choose  from  five  channels. 

•  You  decide  what  news  to  receive. 

■  You’ll  get  breaking  news. 


All  you  need  is  an  RSS  reader. 

Interested?  Visit  us  at  www.erdc.hpc.mil 
or  contact  the  ERDC  MSRC 
Customer  Assistance  Center  at 
1-800-500-4722  or  msrchelp@erdc.hpc.mii 


